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(54) Encoding and decoding system of motion image containing objects of arbitrary shapes 



(57) A motion image encoding apparatus for a cur- 
rent image containing an object uses mesh representa- 
tion for encoding a motion image. The encoding appa- 
ratus includes an object extractor (10) for extracting the 
object contained in the current image from received cur- 
rent image data and outputting object contour data indi- 
cating a contour of the extracted object. A predictive en- 
coder (1 2-1 8) performs a predictive encoding operation 
using the current image data : prestored data of a refer- 
ence image, and control points of meshes which divide 
one of the current image and the reference image, and 
generates motion information involved with the control 
points, differential data between the current image and 



the reference image, and predictive image data. An ob- 
ject difference generator (20, 22) selects differential da- 
ta in an object region among the differential data sup- 
plied from the predictive encoder to encode the selected 
differential data based on the object contour data output 
from the object extractor, and generates the selected dif- 
ferential data and the encoded differential data. An 
adder (24) receives and adds the predictive image data 
output from the predictive encoder and the differential 
data selected in the object difference generator, and up- 
dates the relerence image data prestored in the predic- 
tive encoder using the image data obtained by the ad- 
dition result. 
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Description 

The present invention relates to an encoding and 
decoding system o1 a motion image containing an arbi- 
trary object. 5 

Recently, international standards tor transmitting a 
motion image at a very low bit rate have progressed so 
as to be applied to a video phone or a video conference 
system. To transmit a motion image at a very low rate, 
an object-oriented encoding technique for partitioning a 
meaningful object in the motion image and transmitting 
the partitioned object is widely under study. The object- 
oriented encoding technique is recognized as an essen- 
tial technique in order to transmit a motion image at a 
very low bit rate. In the object-oriented encoding of mo- 
tion image : a motion prediction technique for removing 
temporal correlation with respect to an object is required 
and a more accurate motion prediction technique is 
needed for enhancing an encoding efficiency as well- 
it is an aim of preferred embodiments of the present 
invention to provide an encoding apparatus for a motion 
image containing an arbitrary object in which the arbi- 
trary object extracted from an image is represented as 
meshes and control points of the meshes, are used for 
motion predictive encoding of the image. 

Another aim of embodiments of the present inven- 
tion is to provide a decoding apparatus for decoding da- 
ta encoded in the above encoding apparatus. 

According to a first aspect of the present invention, 
there is provided a motion image encoding apparatus 
for encoding a current image containing an object, the 
motion image encoding a current image containing an 
object, the motion image encoding apparatus compris- 
ing: an object extraction unit for extracting the object 
contained in the current image from received current im- 
age data and outputting object contour data indicating 
a contour of the extracted object; a predictive encoding 
unit for performing a predictive encoding operation us- 
ing the current image data, prestored data of a reference 
image, and control points of meshes which divide one 
of the current image and the reference image, and gen- 
erating motion information involved with the control 
points, differential data between the current image and 
the reference image, and predictive image data; an ob- 
ject difference generator for selecting differential data in 
an object region among the differential data supplied 
from the predictive encoding unit to encode the selected 
differential data based on the object contour data output 
from the object extraction unit ; and generating the se- 
lected differential data and the encoded differential data; 
and an addition unit for receiving and adding the predic- 
tive image data output from the predictive encoding unit 
and the differential data selected in the object difference 
generator, and updating the reference image data 
prestored in the predictive encoding unit using the im- 
age data obtained by the addition result. 

Preferably, said predictive encoding means per- 
forms predictive encoding using spatial transformation 



2 

to generate the predictive image data. 

Said predictive encoding means may comprise: a 
memory for storing reference image data: a mesh gen- 
erator for determining regular meshes having vertices 
located in the object region of the current image as con- 
trol points among the meshes for dividing the entire im- 
age into a predetermined size, based on the object con- 
tour data from said object extraction means and output- 
ting mesh data representing determined regular mesh- 
es; a motion estimation and compensation portion for 
generating the motion information according to the mo- 
tion estimation with respect to the reference image of 
the control points, and the predictive image data whose 
motion of the control points is compensated, using the 
reference image data stored in said memory and the 
mesh data output from said mesh generator; and a dif- 
ferential data generator for generating differential data 
between the current image data and the predictive im- 
age data. 

Preferably said mesh generator comprises: a block 
former for outputting the image block data representing 
a plurality of blocks each having a predetermined size 
obtained by dividing the current image with respect to 
the received current image data; a block selector for se- 
lecting the image block data containing the image data 
of the object region determined by the object contour 
data among the image block data output from said block 
former, and outputting the selected image block data; 
an object image composer for composing the object im- 
age by merging the image blocks of the selected image 
block data; and a mesh composer for determining ver- 
tices contained in the object image constituted by the 
object image composer among vertices of a plurality of 
grids which divide the entire image in size, as control 
points, and determining the grids having the control 
points as rectangular meshes. 

Preferably ; said mesh composer outputs the mesh 
data representing the regular triangular meshes ob- 
tained from the regular rectangular meshes. 

Said regular triangular mesh may be obtained by 
dividing the mesh at a diagonal direction of 45° or -45° 
based on data similarity between the control points of 
each regular rectangular mesh. 

Preferably said predictive encoding means com- 
prises: a memory for storing reference image data; a 
mesh generator for determining irregular meshes hav- 
ing vertices located in the reference image as control 
points using the reference image data stored in said 
memory and outputting mesh data representing deter- 
mined irregular meshes: a motion estimation and com- 
pensation portion for generating the motion information 
according to the motion estimation with respect to the 
current image of the control points, and the predictive 
image data whose motion of the control points is com- 
pensated, using the current image data and the mesh 
data output from said mesh generator; and a differential 
data generator for generating differential data between 
the reference image data and the predictive image data. 
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Said mesh generator may generate the mesh data 
representing the irregular meshes generated by irregu- 
lar triangular mesh representation based on adaptive 
control removal. 

Said object difference generator means preferably 
performs orthogonal transform coding with respect to 
the selected differential data. 

Preferably, said object difference generator means 
comprises: a differential data encoder for selecting the 
differential data belonging to the object region deter- 
mined by the object contour data among the differential 
data output from said predictive encoding means, and 
encoding the selected differential data and the object 
contour data, to output the encoded differential data and 
the encoded object contour data: and a differential data 
decoder for decoding the encoded differential data out- 
put from said differential encoder and outputting the se- 
lected differential data. 

According to a second aspect of the present inven- 
tion, there is also provided a motion image decoding ap- 
paratus for decoding the output of a motion image en- 
coding apparatus, the motion image decoding appara- 
tus comprising: an object contour restoring unit for de- 
coding received encoded object contour data and re- 
storing object contour data indicating a contour of an 
object contained in a current image; a differential data 
decoding unit for decoding the received encoded differ- 
ential data and outputting differential data; a motion 
compensation decoding unit for performing a motion 
compensation operation using the object contour data, 
prestored reference image data, control points of mesh- 
es which divide one of an object contour and a reference 
image, and motion information involved with the control 
points; and an addition unit for adding the predictive im- 
age data selected by the object contour data and the 
differential data, and updating the reference image data 
prestored in the motion compensation decoding unit us- 
ing the image data obtained by the addition result. 

Preferably, said differential data decoding means 
performs orthogonal transform decoding with respect to 
the encoded differential data. 

Said motion compensation decoding means may 
perform motion compensation using spatial transforma- 
tion to generate the predictive image data. 

Preferably, said motion compensation decoding 
means comprises: a memory for storing reference im- 
age data; a mesh generator for determining meshes 
having vertices located in the object region determined 
by the object contour data output from said object con- 
tour restoring means as control points among the mesh- 
es by dividing the entire image into a predetermined 
size, and outputting mesh data representing the deter- 
mined meshes and the object contour data; and a mo- 
tion compensation portion for generating the predictive 
image data whose motion is compensated with respect 
to the reference image of the control points, using the 
reference image data stored in said memory and the 
mesh data. 



Preferably, the mesh data output from said mesh 
generator contains the position information of the verti- 
ces belonging to the determined meshes. 

Said mesh generator preferably outputs mesh data 
5 involving the regular rectangular meshes. 

Said motion image decoding means preferably 
comprises: a memory for storing reference image data; 
a mesh generator for determining meshes for dividing 
the reference image belonging to the object region de- 
10 termined by the object contour data among the refer- 
ence image data stored in said memory and outputting 
mesh data representing determined meshes; and a mo- 
tion estimation and compensation portion for generating 
the predictive image data whose motion is compensated 
15 with respect to the reference image of the control points 
belonging to the meshes, using the received motion in- 
formation, the reference image data stored in the mem- 
ory, and the mesh data. 

Said mesh generator may generate the mesh data 
20 representing the irregular meshes generated by irregu- 
lar triangular mesh representation based on adaptive 
control point removal with respect to the reference im- 
age belonging to the object region. 

For a better understanding of the invention, and i to 
25 show how embodiments of the same may be carried into 
effect, reference will now be made, by way of example, 
to the accompanying diagrammatic drawings, in which: 

Figure 1 is a block diagram showing a motion image 
30 encoding apparatus according to a preferred em- 
bodiment of the present invention; 

Figure 2 is a detailed block diagram of the mesh 
producer of Figure 1 ; 

35 ■ 

Figure 3 is a block diagram showing a motion image 
decoding apparatus for decoding data encoded by 
the Figure 1 apparatus; 

40 Figure 4 is a block diagram showing a motion image 
encoding apparatus according to another preferred 
embodiment of the present invention; and 

Figure 5 is a block diagram showing a motion image 
45 decoding apparatus for decoding data encoded by 
the Figure 4 apparatus. 

Preferred embodiments of the present invention will 
be described in detail with reference to the accompany- 

so ing drawings. 

!n Figure 1 showing a motion image encoding ap- 
paratus according to a preferred embodiment of the 
present invention, an object extractor 10 receives digital 
image data from an external source, and extracts an ar- 

55 bitrary object from a current image represented by the 
received image data. Then, the object extractor 10 gen- 
erates object contour data representing contour infor- 
mation of the extracted object. The object contour data 
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is supplied to a differential data encoder 20 and a de- 
coding apparatus of Figure 3 to be described later. The 
object contour data contains spatial positions of pixels 
which determine a contour of the object or spatial posi- 
tions of segments composed of the pixels. The object 5 
extractor 10 outputs the object contour data and the re- 
ceived current image data to a mesh generator 12. The 
mesh generator 12 discriminates an object region from 
the current image based on the object contour data, and 
performs a signal processing for regular mesh represen- 
tation with respect to the image in the object region. The 
detailed structure and operation of the mesh generator 
12 will be described with reference to Figure 2. 

A block former 121 in the mesh generator 12 re- 
ceives the current image data and the object contour 
data output from the object extractor 10. The block 
former 1 21 divides the current image represented by the 
received image data into image blocks each having a 
predetermined size, and supplies image block data rep- 
resenting image blocks and the object contour data to 
a block selector 123. The block selector 123 selects im- 
age blocks containing the image data in the object re- 
gion among the entire image blocks constituting the cur- 
rent image based on the object contour data. The data 
representing the selected image blocks and the object 
contour data is output to an object image composer 125. 
Here, image blocks having only image data which do 
not belong to the object region, are not selected. There- 
fore, the data representing the image blocks which are 
not selected is not supplied tothe object image compos- 
er 125. The object image composer 125 which receives 
the data of the selected image blocks merges the se- 
lected image blocks and supplies the data representing 
the object image generated from the merging result and 
the object contour data to a mesh composer 127. The 
mesh composer 1 27 receiving the object image data di- 
vides the object image into a regular rectangular mesh 
or a regular triangular mesh. 

In the case that an image is divided using regular 
rectangular meshes : the mesh composer 127 divides 
the object image into grids each having a predetermined 
size different from a block size. Here, a grid has a size 
smaller than that of a block. Then, the mesh composer 
1 27 determines vertices contained in the object region 
among the vertices of the grids obtained by the division 
as control points, and determines grids having the con- 
trol points as regular rectangular meshes. The mesh 
composer 127 outputs rectangular mesh data contain- 
ing the image data involving the control points of the reg- 
ular rectangular meshes to a motion estimation and 
compensation portion 14 of Figure 1. Here, the image 
data involving each control point includes the position 
of a control point and a gray value at the position ol the 
control point, and the positions of the pixels having po- 
sitions adjacent to the control point and the gray values. 
The mesh composer 127 also outputs the current image 
data to a difference value calculator 18. 

In the case that an image is divided using regular 
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triangular meshes : the mesh composer 127 divides 
each regular rectangular mesh obtained through the 
above process into two regular triangular meshes based 
on similarity ol the image data in the mesh. To judge 
similarity of the image data of each regular rectangular 
mesh, the mesh composer 127 compares a difference 
value between pixel values corresponding to two control 
points which are located at a diagonal direction of 45° 
in the regular rectangular mesh, with a difference value 
between pixel values corresponding to two control 
points located at a diagonal direction of -45° therein in 
size. The mesh composer 127 divides the rectangular 
mesh in the diagonal direction corresponding to the 
smaller difference value. As a result, two regular trian- 
gular meshes are generated every rectangular mesh. 
The mesh composer 127 outputs the triangular mesh 
data containing the image data involving the control 
points of the regular triangular meshes tothe motion es- 
timation and compensation portion 1 4. The mesh com- 
poser 1 27 outputs the current image data to a difference 
value calculator 18 as well. 

The motion estimation and compensation portion 
14 performs a motion estimation and compensation us- 
ing the reference image data stored in the memory 16 
and the mesh data involving the object image in the cur- 
rent image received from the mesh former 12. The mo- 
tion estimation and compensation portion 14 first com- 
pares in magnitude the image data involved in the con- 
trol points of each mesh with the reference image data 
stored in the memory 16. Each control point corre- 
sponds to a single pixel ideally. That is, a control point 
is expressed by a spatial position of a pixel and a gray 
value. However, it is actually difficult to exactly find out 
a position having image data similar to the control point 
in the reference image by only pixel to pixel comparison. 
Thus, the image data involving each control point is im- 
age data which is involved in a pixel corresponding to 
the control point and a plurality of pixels neighbouring 
the pixel corresponding to the control point, as de- 
scribed in the mesh composer 127. The motion estima- 
tion and compensation portion 14 determines a position 
in the reference image having the image data which is 
most similar to the image data of the control point based 
on the data magnitude comparison result. Then, the mo- 
tion estimation and compensation portion 14 deter- 
mines a motion vector representing movement between 
a control point and a corresponding position in the ref- 
erence image. If the motion vectors are determined with 
respect to all the control points, the motion estimation 
and compensation portion 14 performs spatial transfor- 
mation such as image warping using the determined 
motion vectors : and generates a predictive image with 
respect to the current image. The data representing the 
predictive image is output to the difference value calcu- 
lator 18 and the adder 24. The motion information rep- 
resenting the motion vectors corresponding to all the 
control points is supplied to the Figure 3 apparatus. 
Since the above -described spatial transformation is well 
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known in the field of a digital image processing, the de- 
tailed description thereof will be omitted. 

The difference value calculator 18 calculates differ- 
ence values between the current image data received 
from the mesh generator 12 and the predicitve image 5 
data output from the motion estimation and compensa- 
tion portion 14. The difference value calculation is per- 
formed between the pixels having corresponding posi- 
tions with respect to the current image and the predictive 
image, and the resultant difference value data is output 10 
to a differential data encoder 20. The differential data 
encoder 20 encodes difference value data in the object 
region determined by the object contour data of the ob- 
ject extractor 10 among the difference value data sup- 
plied from the difference value calculator 1 8. If the data *5 
encoded by the differential data encoder 20 is defined 
as difference value data in the object region, only differ- 
ence value data involving the object of the current image 
is encoded. Accordingly, the differential data encoder 20 
can perform a more efficient encoding operation with re- 20 
spect to the current image. The differential data encoder 
20 encodes the object contour data received from the 
object extractor 1 0 as well. To encode the difference val- 
ue data and the object contour data in the object region, 
the differential data encoder 20 uses a well-known or- 25 
thogonal transform coding method including discrete 
cosine transform (DCT) and so on. The encoded object 
contour data, the encoded difference value data and the 
motion information are transmitted to a decoding appa- 
ratus of Figure 3 to be described later via a transmission 30 
channel (not shown), or are recorded on a storage me- 
dium used in the decoding apparatus of Figure 3. 

A differential data decoder 22 receiving the encod- 
ed difference value data from the differential data en- 
coder 20 restores the difference value data via a reverse 3S 
procedure of the signal processing of the differential da- 
ta encoder 20. The adder 24 adds the predicitve image 
data output from the motion estimation and compensa- 
tion portion 14 and the difference value data of the dif- 
ferential data decoder 22, and outputs the added result 40 
to the memory 1 6. The data output from the adder 24 is 
data of the current image from which motion estimation 
and compensation has been performed, and is stored 
in the memory 16 to be used as reference image data 
for motion estimation and motion compensation with re- 45 
spect to a next image. 

A motion image decoding apparatus shown in Fig- 
ure 3 receives the encoded difference value data, the 
encoded object contour data and the motion information 
generated by the Figure 1 apparatus. An object contour so 
restorer 30 decodes the encoded object contour data. 
The object contour data is output to a mesh generator 
34. The mesh generator 34 receiving the object contour 
data divides the entire image into grids each having a 
predetermined size, determines vertices existing in the ss 
object region determined by the object contour data as 
control points among the vertices of the grids obtained 
by the division, and determines the grids having the con- 
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trol points as rectangular meshes. In the case that the 
mesh generator 12 of Figure 1 is designed to generate 
mesh data representing the rectangular meshes, the 
mesh generator 34 also generates mesh data repre- 
senting rectangular meshes. Meanwhile, in the case 
that the mesh generator 12 of Figure 1 is designed to 
generate the mesh data involvoing the triangular mesh- 
es, the mesh generator 34 generates mesh data involv- 
ing the triangular meshes, the mesh generator 34 sup- 
plies the generated mesh data to a motion compenstor 
38 and supplies the object contour data to an adder 36. 

Meanwhile, a differential data decoder 32 decodes 
the encoded difference value data generatd by the Fig- 
ure 1 apparatus. The difference value data is output to 
the adder 36. The motion compensator 38 receives the 
mesh data output from the mesh generator 34 and the 
motion information generated by the Figure 1 appara- 
tus. The motion compensator 38 generates predictive 
image data using control points contained in the mesh 
data corresponding to a current image, motion vectors 
corresponding to alt the control points and contained in 
motion information, and the reference image data stored 
in the memory 40. To generate the predictive image da- v 
ta, the motion compensator 38 uses a motion vector cor- * 
responding to each control point to find out a position in * 
the reference image which is most similar to the control *■ * 
point, and then uses spatial transformation such as im- ~ 
age warping to generate predictive image data for a cur- U 
rent image. The motion compensator 38 outputs the 
generated predictive image data to an adder 36. 

The adder 36 receives the predictive image data 
output from the motion compensator 38 and the differ- ft 
ence value data output from the differential data decod- h% - 
er 32. The adder 36 adds only predictive image data in • « 
the object region determined by the object contour data ■< 
supplied from the mesh generator 34 among the entire ' 
predictive image data to the difference value data cor- 
responding thereto. By doing so, image data with re- 
spect to an object in the current image can be restored. 
The output data of the adder 36 is stored in the memory 
40 to be used as reference image data for motion com- 
pensation of a next image. 

Figure 4 is a block diagram showing a motion image 
encoding apparatus according to another preferred em- 
bodiment of the present invention. Since the blocks of 
Figure 4 having the same reference numerals as those 
of the Figure 1 blocks perform the same functions as 
those of the corresponding blocks of figure 1, the de- 
tailed description thereof will be omitted. The Figure 1 
apparatus uses the meshes obtained from the current 
image and the reference image data stored in the mem- 
ory 1 6 to generate predictive image data. On the con- 
trary, the Figure 4 apparatus uses meshes obtained 
from reference image data stored in a memory 48 and 
externally input current image' data to generate predic- 
tive image data. 

The external current image data is input to the ob- 
ject extractor 10 and a motion estimation and compen- 
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sation portion 44. The object extractor 10 generates ob- 
ject contour data using the current image data. The ob- 
ject contour data is output to the differential data encod- 
er 20. A mesh generator 42 reads the reference image 
data stored in the memory 48 and divides the entire ref- 
erence into irregular meshes. For such division, the 
mesh generator 42 divides the reference image into reg- 
ular rectangular meshes and divides the regular rectan- 
gular meshes into regular triangular meshes again. 
Since the method for generation of the regular rectan- 
gular mesh and the regular triangular mesh is same as 
a signal processing of the Figure 1 mesh generator 12, 
the detailed description thereof will be omitted. 

Another function of the mesh generator 42 gener- 
ates an irregular mesh from regular rectangular meshes 
or regular triangular meshes. The procedure of gener- 
ating the irregular mesh is as follows. The mesh gener- 
ator 42 removes one selected among the control points 
of the regular triangular meshes, and performs triangu- 
lation with respect to the region from which the control 
point is removed. The triangular meshes generated by 
the triangulation have irregular triangular shapes. Such 
control point removal and triangulation repeats until the 
number of the remaining control points is identical to a 
predetermined value. To remove a particular control 
point, a quantified difference value between image de- 
scriptiveness obtained when the control point has been 
removed from a support region of the control point and 
that when the former has not been removed. Here, the 
support region is a region surrounded by neighbouring 
control points to a corresponding control point and lines 
connecting the neighbouring control points. A control 
point corresponding to the smallest difference value 
among the quantified difference values of the image de- 
scriptiveness does little to contribute for the image de- 
scriptiveness, to accordingly be removed. The irregular 
mesh generation technique is disclosed in a paper en- 
titled "Irregular Triangular Mesh Representation Based 
on Adaptive Control Point Removal" published in SPIE's 
1 996 symposium on Visual Communications and Image 
Processing by Kang W. Chun, Byungwoo Jean and Jae 
M. Jo. The mesh generator 42 outputs information about 
the finally generated irregular triangular meshes : that is, 
the irregular mesh data representing the remaining con- 
trol points and the irregular meshes surrounded by the 
control points, to a motion estimation and compensation 
portion 44. The mesh generator 42 outputs the refer- 
ence image data read from the memory 48 to a differ- 
ence value calculator 46 as well. 

The motion estimation and compensation portion 
44 receives the externally supplied current image data 
and the irregular mesh data of the mesh generator 42, 
and reads the reference image data stored in the mem- 
ory 48. The motion estimation and compensation por- 
tion 44 determines corresponding positions in the cur- 
rent image having the image data which is most similar 
to the control points of the irregular meshes. After the 
corresponding positions are determined, the motion es- 



timation and compensation portion 44 generates motion 
vectors between the control points and the correspond- 
ing positions and uses spatial transformation to gener- 
ate a predictive image using the current image. The pre- 
5 dictive image data is supplied to the difference value cal- 
culator 46 and the adder 24, and the motion information 
representing the motion vectors are transmitted via a 
transmission channel or recorded on a storage medium 
to be used for a decoding apparatus. 
10 The difference value calculator 46 generates differ- 
ence value data between the reference image data re- 
ceived from the mesh generator 42 and the predictive 
image data from the motion estimation and compensa- 
tion portion 44, to supply the difference value data to a 
is differential data encoder 20. The differential data encod- 
er 20 encodes the object contour data output from the 
object extractor 10 and the difference value data output 
from the difference value calculator 46. The differential 
data encoder 20 processes the input data in the same 
manner as those of the corresponding blocks of Figure 
1. Thus, the encoded data output from the differential 
data encoder 20 includes data obtained by encoding the 
difference value data in the object region and the en- 
coded object contour data. The encoded data is trans- 
mitted to a receiver via a transmission channel (not 
shown), or is recorded on a recording medium (not 
shown). The encoded difference value data among the 
encoded data is transmitted to the differential data de- 
coder 22. 

The differential data decoder 22 decodes the differ- . 
ence value data encoded in the differential data encoder 
20, to supply the decoded difference value data to the 
adder 24. The adder 24 adds the predictive image data 
supplied from the motion estimation and compensation 
portion 44 and the difference value data of the differen- 
tial data decoder 22 : and the resultant data is stored in 
the memory 49, to be used as reference image data for 
motion estimation and motion compensation for a next 
image. 

Figure 5 shows a motion image decoding apparatus 
corresponding to the Figure 4 apparatus. Since blocks 
having the same reference numerals as those of the Fig- 
ure 3 blocks among the blocks shown in Figure 5 have 
the same functions as those of the Figure 3 blocks, the 
detailed description thereof will be omitted. The encod- 
ed object contour data is input to an object contour re- 
storer 30 and the motion information is input to a motion 
compensator 52. Also, the encoded difference value da- 
ta is supplied to the differential data decoder 32. The 
differential data decoder 32 decodes the encoded dif- 
ference value data to output the decoded difference val- 
ue data to an adder 56. The object contour restorer 30 
decodes the encoded object contour data to output the 
decoded object contour data to a mesh generator 51. 
The mesh generator 51 reads the reference image data 
stored in a memory 54, and divides the object region in 
the reference image determined by the object contour 
data via the same signal processing as that of the Figure 
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4 mesh generator 42, into irregular meshes. The mesh 
data output from the mesh generator 51 is supplied to 
the motion compensator 52. The motion compensator 
52 uses the mesh data, the motion information and the 
reference image data stored in the memory 54 to gen- s 
erate the predictive image data. The adder 56 adds the 
predictive image data and the difference value data out- 
put from the differential data decoder 32. The resultant 
current image is stored in the memory 54 to be used as 
reference image data for motion compensation of a next 10 
image. 

As described above, the encoding and decoding 
system according to the present invention performs mo- 
tion estimation and motion compensation based on 
mesh representation with respect to an arbitrary object is 
contained in an image. As a result, a more efficient en- 
coding can be performed compared with a case when 
motion estimation and motion compensation is per- 
formed with respect to the entire image. Thus, an appa- 
ratus appropriate for applications requiring a very low 20 
bit rate can be designed. 

While only certain embodiments of the invention 
have been specifically described herein, it will be appar- 
ent that numerous modifications may be made thereto 
without departing from the scope ol the invention. 25 

The reader's attention is directed to all papers and 
documents which are filed concurrently with or previous 
to this specification in connection with this application 
and which are open to public inspection with this spec- 
ification, and the contents of all such papers and docu- 30 
ments are incorporated herein by reference. 

All of the features disclosed in this specification (in- 
cluding any accompanying claims, abstract and draw- 
ings), and/or all of the steps of any method or process 
so disclosed, may be combined in any combination: ex- 35 
cept combinations where at least some of such features 
and/or steps are mutually exclusive. 

Each feature disclosed in this specification (includ- 
ing any accompanying claims, abstract and drawings), 
may be replaced by alternative features serving the *o 
same, equivalent or similar purpose, unless expressly 
stated otherwise. Thus, unless expressly staled other- 
wise, each feature disclosed is one example only of a 
generic series of equivalent or similar features. 

The invention is not restricted to the details of the 45 
foregoing embodiment(s). The invention extends to any 
novel one, or any novel combination, of the features dis- 
closed in this specification (including any accompanying 
claims, abstract and drawings); or to any novel one, or 
any novel combination, of the steps of any method or 50 
process so disclosed. 



Claims 

1 . A motion image encoding apparatus for encoding a 
current image containing an object, the motion im- 
age encoding apparatus comprising: 



object extraction means (10) for extracting the 
object contained in the current image from re- 
ceived current image data and outputting object 
contour data indicating a contour of the extract- 
ed object; 

predictive encoding means (12, 14, 16, 18; 42, 
44, 46, 48) for performing a predictive encoding 
operation using the current image data, 
p restored data of a reference image, and con- 
trol points of meshes which divide one of the 
current image and the reference image, and 
generating motion information relating to the 
control points, differential data between the cur- 
rent image and the reference image, and pre- 
dictive image data; 

object difference generator means (20, 22) for 
selecting differential data in an object region 
among the differential data supplied from the 
predictive encoding means (12 : 14, 16, 18; 42, 
44, 46, 48) to encode the selected differential 
data based on the object contour data output 
from the object extraction means (10), and out- 
putting the selected differential data and the en- 
coded differential data; and 

adder means (24) for receiving and adding the 
predictive image data output from the predictive 
encoding means and the selected differential 
data output from the object difference genera- 
tor means, and updating the reference image 
data prestored in the predictive encoding 
means using the image data obtained by the 
addition result. 

The motion image encoding apparatus according to 
claim 1, wherein said predictive encoding means 
(12-18; 42-48) performs predictive encoding using 
spatial transformation to generate the predictive im- 
age data. 

The motion image encoding apparatus according to 
claim 1 or 2, wherein said predictive encoding 
means (12-18) comprises: 

a memory (16) for storing reference image da- 
ta; 

a mesh generator (12) for determining regular 
meshes having vertices located in the object re- 
gion of the current image as control points 
among the meshes for dividing the entire image 
into a predetermined size, based on the object 
contour data from said object extraction means 
(1 0) and outputting mesh data representing de- 
termined regular meshes; 
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a motion estimation and compensation portion 
(14) for generating the motion information ac- 
cording to the motion estimation with respect to 
the reference image of the control points, and 
the predictive image data whose motion of the 
control points is compensated, using the refer- 
ence image data stored in said memory (16) 
and the mesh data output from said mesh gen- 
erator (12); and 

a differential data generator (18) for generating 
differential data between the current image da- 
ta and the predictive image data. 

The motion image encoding apparatus according to 
claim 3, wherein said mesh generator (12) compris- 
es: 

a block former (121) for outputting the image 
block data representing a plurality of blocks 
each having a predetermined size obtained by 
dividing the current image with respect to the 
received current image data; 

a block selector (123) for selecting the image 
block data containing the image data of the ob- 
ject region determined by the object contour da- 
ta among the image block data output from said 
block former (121), and outputting the selected 
image block data; 

an object image composer (125) for composing 
the object image by merging the image blocks 
of the selected image block data; and 

a mesh composer (127) for determining verti- 
ces contained in the object image constituted 
by the object image composer (125) among 
vertices of a plurality of grids which divide the 
entire image in size, as control points, and de- 
termining the grids having the control points as 
rectangular meshes. 

The motion image encoding apparatus according to 
claim 4, wherein said mesh composer (1 27) outputs 
the mesh data representing the regular triangular 
meshes obtained from the regular rectangular 
meshes. 

The motion image encoding apparatus according to 
claim 5 : wherein said regular triangular mesh is ob- 
tained by dividing the mesh at a diagonal direction 
of 45° or -45° based on data similarity between the 
control points of each regular rectangular mesh. 

The motion image encoding apparatus according to 
claim 1, wherein said predictive encoding means 
comprises: 



a memory (48) for storing reference image da- 
ta; 

a mesh generator (42) for determining irregular 
5 meshes having vertices located in the refer- 

ence image as control points using the refer- 
ence image data stored in said memory (48) 
and outputting mesh data representing deter- 
mined irregular meshes; 

10 

a motion estimation and compensation portion 
(44) for generating the motion information ac- 
cording to the motion estimation with respect to 
the current image of the control points, and the 
is predictive image data whose motion of the con- 

trol points is compensated, using the current 
image data and the mesh data output from said 
mesh generator (42); and 

20 a differential data generator (46) for generating 

differential data between the reference image 
data and the predictive image data. 

8. The motion image encoding apparatus according to 
25 claim 7, wherein said mesh generator (42) gener- 
ates the mesh data representing the irregular mesh : 
es generated by irregular triangular mesh represen- 
tation based on adaptive control removal. 

30 9. The motion image encoding apparatus according to 
any of claims 1 to 8, wherein said object difference 
generator means performs orthogonal transform 
coding with respect to the selected differential data. 

35 10. The motion image encoding apparatus according to 
any of claims 1 to 9, wherein said object difference 
generator means comprises: 

a differential data encoder (20) for selecting the 
40 differential data belonging to the object region 

determined by the object contour data among 
the differential data output from said predictive 
encoding means, and encoding the selected 
differential data and the object contour data, to 
45 output the encoded differential data and the en- 

coded object contour data; and 

a differential data decoder (22) for decoding the 
encoded differential data output from said dif- 
so ferential encoder and outputting the selected 

differential data. 

11. A motion image decoding apparatus for the output 
of a motion image encoding apparatus, the motion 
55 image decoding apparatus comprising: 

object contour restoring means (30) for decod- 
ing received encoded object contour data and 
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restoring object contour data indicating a con- 
tour of an object contained in a current image; 

differential data decoding means (32) for de- 
coding the received encoded differential data 5 
and outputting differential data; 

motion compensation decoding means (34, 52, 
54) for performing a motion compensation op- 
eration using data of the object contour, io 
prestored data of a reference image, control 
points of meshes which divide one of the object 
contour and the reference image, and motion 
information involved with the control points; 
and is 

adder means for adding the predictive image 
data selected by the object contour data and 
the differential data, and updating the reference 
image data prestored in the motion compensa- 20 
tion decoding means (34, 38, 40; 51, 52, 54) 
using the image data obtained by the addition 
result. 

12. The motion image decoding apparatus according to 25 
claim 11 or 1 2, wherein said differential data decod- 
ing means (32) performs orthogonal transform de- 
coding with respect to the encoded differential data. 

1 3. The motion image decoding apparatus according to 30 
claim 11, 1 2 or 1 3, wherein said motion compensa- 
tion decoding means (34, 33, 40; 51. 52, 54) per- 
forms motion compensation using spatial transfor- 
mation to generate the predictive image data. 

35 

14. The motion image decoding apparatus according to 
claim 1 1 , wherein said motion compensation decod- 
ing means (34, 52, 54) comprises: 

a memory (40) for storing reference image da- 40 
ta; 

a mesh generator (34) for determining meshes 
having vertices located in the object region de- 
termined by the object contour data output from 45 
said object contour restoring means (30) as 
control points among the meshes by dividing 
the entire image into a predetermined size, and 
outputting mesh data representing the deter- 
mined meshes and the object contour data; and so 

a motion compensation portion (38) for gener- 
ating the predictive image data whose motion 
is compensated with respect to the reference 
image of the control points, using the received 5 $ 
motion information, the reference image data 
stored in said memory (40) and the mesh data. 



32 A2 16 

1 5. The motion image decoding apparatus according to 
claim 14, wherein the mesh data output from said 
mesh generator (34) contains the position informa- 
tion of the vertices belonging to the determined 
meshes. 

16. The motion image decoding apparatus according to 
claim 1 4, wherein said mesh generator (34) outputs 
mesh data involving the regular rectangular mesh- 
es. 

17. The motion image decoding apparatus according to 
claim 11, wherein said motion image decoding 
means (51, 52, 54) comprises: 

a memory (54) for storing reference image da- 
ta; 

a mesh generator (51) for determining meshes 
for dividing the reference image belonging to 
the object region determined by the object con- 
tour data among the reference image data 
stored in said memory (54) and outputting 
mesh data representing determined meshes; 
and 

a motion estimation and compensation portion 
(52) for generating the predictive image data 
whose motion is compensated with respect to 
the reference image of the control points be- 
longing to the meshes, using the received mo- 
tion information, the reference image data 
stored in the memory, and the mesh data. 

18. The motion image decoding apparatus according to 
claim 17, wherein said mesh generator (51) gener- 
ates the mesh data representing the irregular mesh- 
es generated by irregular triangular mesh represen- 
tation based on adaptive control point removal with 
respect to the reference image belonging to the ob- 
ject region. 
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